198 research outputs found
Putting the Horse Before the Cart:A Generator-Evaluator Framework for Question Generation from Text
Automatic question generation (QG) is a useful yet challenging task in NLP.
Recent neural network-based approaches represent the state-of-the-art in this
task. In this work, we attempt to strengthen them significantly by adopting a
holistic and novel generator-evaluator framework that directly optimizes
objectives that reward semantics and structure. The {\it generator} is a
sequence-to-sequence model that incorporates the {\it structure} and {\it
semantics} of the question being generated. The generator predicts an answer in
the passage that the question can pivot on. Employing the copy and coverage
mechanisms, it also acknowledges other contextually important (and possibly
rare) keywords in the passage that the question needs to conform to, while not
redundantly repeating words. The {\it evaluator} model evaluates and assigns a
reward to each predicted question based on its conformity to the {\it
structure} of ground-truth questions. We propose two novel QG-specific reward
functions for text conformity and answer conformity of the generated question.
The evaluator also employs structure-sensitive rewards based on evaluation
measures such as BLEU, GLEU, and ROUGE-L, which are suitable for QG. In
contrast, most of the previous works only optimize the cross-entropy loss,
which can induce inconsistencies between training (objective) and testing
(evaluation) measures. Our evaluation shows that our approach significantly
outperforms state-of-the-art systems on the widely-used SQuAD benchmark as per
both automatic and human evaluation.Comment: 10 pages, The SIGNLL Conference on Computational Natural Language
Learning (CoNLL 2019
Neural Architecture for Question Answering Using a Knowledge Graph and Web Corpus
In Web search, entity-seeking queries often trigger a special Question
Answering (QA) system. It may use a parser to interpret the question to a
structured query, execute that on a knowledge graph (KG), and return direct
entity responses. QA systems based on precise parsing tend to be brittle: minor
syntax variations may dramatically change the response. Moreover, KG coverage
is patchy. At the other extreme, a large corpus may provide broader coverage,
but in an unstructured, unreliable form. We present AQQUCN, a QA system that
gracefully combines KG and corpus evidence. AQQUCN accepts a broad spectrum of
query syntax, between well-formed questions to short `telegraphic' keyword
sequences. In the face of inherent query ambiguities, AQQUCN aggregates signals
from KGs and large corpora to directly rank KG entities, rather than commit to
one semantic interpretation of the query. AQQUCN models the ideal
interpretation as an unobservable or latent variable. Interpretations and
candidate entity responses are scored as pairs, by combining signals from
multiple convolutional networks that operate collectively on the query, KG and
corpus. On four public query workloads, amounting to over 8,000 queries with
diverse query syntax, we see 5--16% absolute improvement in mean average
precision (MAP), compared to the entity ranking performance of recent systems.
Our system is also competitive at entity set retrieval, almost doubling F1
scores for challenging short queries.Comment: Accepted to Information Retrieval Journa
Leveraging continuous integration in space avionics - a design using declarative build automation paradigm
There are several benefits when Continuous Integration (CI) is adopted for a software development project. This provides for a mechanism to reduce the burden on developers during the build and test of the developed software, as well as help release the product on-time. Other benefits include capturing errors quite early in the development cycle, easier integration at defined intervals over the course of software development, and faster, comprehensive feedback to developers. However, in an embedded domain, adopting CI is a challenging activity. If the project size and complexity is high, there will be a large number of activities which need to be covered in the CI workflow. Not all tools used in software development provide seamless interfaces to the CI tool. There is a need to design the interface framework which can quickly grow to be complex and time consuming.
An effective CI workflow follows a set of best practices. Build automation is one of them. The existing literature does not provide comprehensive information to address the effect that the build automation tools have on the design and implementation of a CI framework in an embedded avionics domain. Tools like GNU Make and Apache Ant are primarily used for the build and test stages of development. However, these build tools are imperative in nature. As the build logic increases in complexity, the conciseness of build scripts reduces. The build runtimes should also not be large as the feedback cycle time would be longer.
This study aims to design a CI workflow for a space satellite On-Board Software(OBSW) development project. The objective is to bring out the limitations and challenges of using a conventional imperative build approach during the set-up of a CI framework for the project. The proposal is to adopt a build tool which is based on declarative build paradigms and provide for mechanisms to easily integrate with CI tools. This study is carried out as an action research (AR) with study results expressed as quantitative or qualitative metrics. A prototypical CI chain is implemented with a Jenkins CI server and Gradle as the primary build tool. Parameters such as performance, maintenance complexity of build logic, and features such as integration to a CI tool, reproducible builds are investigated
GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning
Large scale machine learning and deep models are extremely data-hungry.
Unfortunately, obtaining large amounts of labeled data is expensive, and
training state-of-the-art models (with hyperparameter tuning) requires
significant computing resources and time. Secondly, real-world data is noisy
and imbalanced. As a result, several recent papers try to make the training
process more efficient and robust. However, most existing work either focuses
on robustness or efficiency, but not both. In this work, we introduce Glister,
a GeneraLIzation based data Subset selecTion for Efficient and Robust learning
framework. We formulate Glister as a mixed discrete-continuous bi-level
optimization problem to select a subset of the training data, which maximizes
the log-likelihood on a held-out validation set. Next, we propose an iterative
online algorithm Glister-Online, which performs data selection iteratively
along with the parameter updates and can be applied to any loss-based learning
algorithm. We then show that for a rich class of loss functions including
cross-entropy, hinge-loss, squared-loss, and logistic-loss, the inner discrete
data selection is an instance of (weakly) submodular optimization, and we
analyze conditions for which Glister-Online reduces the validation loss and
converges. Finally, we propose Glister-Active, an extension to batch active
learning, and we empirically demonstrate the performance of Glister on a wide
range of tasks including, (a) data selection to reduce training time, (b)
robust learning under label noise and imbalance settings, and (c) batch-active
learning with several deep and shallow models. We show that our framework
improves upon state of the art both in efficiency and accuracy (in cases (a)
and (c)) and is more efficient compared to other state-of-the-art robust
learning algorithms in case (b)
- …